Configuration and operation of display devices including device management

ABSTRACT

Device management includes receiving an image including one or more display devices, the image including contextual data associated with environmental information surrounding the one or more display devices. An updated image of the image of the one or more display devices is displayed. The updated image projects a view of the one or more display devices from a desired perspective.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority benefit of U.S. Provisional PatentApplication Ser. No. 62/305,938, filed Mar. 9, 2016, which isincorporated herein by reference in its entirety.

TECHNICAL FIELD

One or more embodiments generally relate to display devices and, inparticular, to configuring and controlling display devices includingdevice management and arrangement.

BACKGROUND

As the ecosystem of devices and displays grows, there is often a needfor users to transfer content from one device to the other. In spite ofthe availability of communication systems such as BLUETOOTH® and Wi-Fi,the interaction interfaces used to achieve these operations continue tobe very unintuitive. Typically, a user would need to find and rememberthe name of the display, its representation in a network/networkidentification (ID), look it up from a list of names and select thatname to connect to the display. This issue is compounded when the userhas to curate a cluster of displays, making it very hard to know whichdisplay they are currently working with.

SUMMARY

One or more embodiments generally relate to configuring and controllingdisplay devices including display device management and arrangement. Inone embodiment, a method provides for device management that includesreceiving an image including one or more display devices, the imageincluding contextual data associated with environmental informationsurrounding the one or more display devices. An updated image of theimage of the one or more display devices is displayed. The updated imageprojects a view of the one or more display devices from a desiredperspective.

In another embodiment, a system for device management includes a memorystoring instructions and at least one processor executing theinstructions including a process configured to: receive an imageincluding one or more display devices, the image including contextualdata associated with environmental information surrounding the one ormore display devices, and display an updated image of the image of theone or more display devices, the updated image projecting a view of theone or more display devices from a desired perspective.

In one embodiment non-transitory processor-readable medium that includesa program that when executed by a processor perform a method comprising:receiving an image including one or more display devices the imageincluding contextual data associated with environmental informationsurrounding the one or more display devices. An updated image of theimage of the one or more display devices is displayed. The updated imageprojects a view of the one or more display devices from a desiredperspective.

These and other aspects and advantages of one or more embodiments willbecome apparent from the following detailed description, which, whentaken in conjunction with the drawings, illustrate by way of example theprinciples of the one or more embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

For a fuller understanding of the nature and advantages of theembodiments, as well as a preferred mode of use, reference should bemade to the following detailed description read in conjunction with theaccompanying drawings, in which:

FIG. 1 shows a schematic view of a communications system, according toan embodiment;

FIG. 2 shows a block diagram of architecture for a system including amobile device including a content playback module, according to anembodiment;

FIGS. 3A-D show an example of continuous lock of virtual overlays on anelectronic device for a display device, according to an embodiment;

FIG. 4A shows an example of a cluster of multiple display devices,according to an embodiment;

FIG. 4B shows a mobile device capturing a layout of the cluster ofmultiple display devices, according to an embodiment;

FIGS. 5A-F show a sequence diagram for an example of managing the imagecontent displayed on a cluster of multiple display devices, according toan embodiment;

FIGS. 6A-D show a sequence diagram of an example for managing the imagecontent displayed on a display device by adjusting scale, according toan embodiment;

FIGS. 7A-D show diagrams of an example for managing the image contentdisplayed on multiple display devices by spreading a single imagecontent across the multiple display devices in a spatially cohesivemanner, according to an embodiment;

FIGS. 8A-D show diagrams of an example for managing the image contentdisplayed on multiple display devices by overlaying on top of a selectedimage content during the image content selection process so that theimage content is cropped over a whole group of display devices includingthe spaces among them, according to an embodiment;

FIGS. 9A-F show diagrams of an example for managing the image contentdisplayed on display devices by spreading a single image content acrossmultiple display devices based on regions of interest, according to anembodiment;

FIGS. 10A-D show diagrams of an example for managing the image contentdisplayed on display devices by changing the relative relationship ofthe image content displayed on a display device, according to oneembodiment;

FIGS. 11A-F show a sequence diagram of an example for sharing imagecontent shown in any of the multiple display devices, according to oneembodiment;

FIGS. 12A-B show an example illustration of finger movement drag anddrop for moving image content between display devices, according to oneembodiment;

FIGS. 13A-B show an example illustration of finger movement drag anddrop for applying predetermined filters or filters based on ambientcolors for display devices, according to one embodiment;

FIGS. 14A-C show an example illustration of selecting colors from theimage content being displayed to be used as the mat color for displaydevices, according to one embodiment;

FIG. 15 shows an example illustration of grouping of display devices inmultiple areas, according to one embodiment;

FIGS. 16A-C show an example illustration of managing groups of displaydevices using a mobile device, according to one embodiment;

FIGS. 17A-B show an example illustration of an example templaterecommendation that measures the display devices and providesrecommendations of various configurations for multiple display devices,according to one embodiment;

FIGS. 18A-B show an example illustration of an example templaterecommendation for an overall canvas layout for multiple displaydevices, according to one embodiment;

FIG. 19 shows an example illustration of different viewing perspectivesbased on relative location of viewer, for multiple display devices;

FIGS. 20A-D show an example illustration of determining the position ofan ideal view point, based on the size, shape and overall layout ofmultiple display devices, according to one embodiment;

FIGS. 21A-D show an example illustration of capturing a frame, analyzingthe prominent colors in a room and displaying these colors as ascrollable list and presenting images based on the list on an electronicdevice, according to one embodiment;

FIGS. 22A-C show an example illustration of assessing the color of awall surrounding a display frame and using that information to colormatch the mat of a display device, according to one embodiment;

FIGS. 23A-C show an example illustration of identifying repeatingpatterns, and digitally synthesizing the patterns to provide a similartransparent mat effect for a display device, according to oneembodiment;

FIGS. 24A-C shows an example of customizing matting for imaging on adisplay device, according to an embodiment;

FIGS. 25A-B shows another example of customizing matting for imaging ona display device, according to an embodiment;

FIG. 26 shows a block diagram of a process for overlaying visual effectson a captured image, according to one embodiment;

FIG. 27 shows a block diagram of a process for spreading an image overmultiple display devices, according to one embodiment;

FIG. 28 shows a block diagram of a process for managing display devices,according to one embodiment; and

FIG. 29 is a high-level block diagram showing an information processingsystem comprising a computing system implementing one or moreembodiments.

DETAILED DESCRIPTION

The following description is made for the purpose of illustrating thegeneral principles of one or more embodiments and is not meant to limitthe inventive concepts claimed herein. Further, particular featuresdescribed herein can be used in combination with other describedfeatures in each of the various possible combinations and permutations.Unless otherwise specifically defined herein, all terms are to be giventheir broadest possible interpretation including meanings implied fromthe specification as well as meanings understood by those skilled in theart and/or as defined in dictionaries, treatises, etc.

One or more embodiments provide for easy content curation for a clusterof display devices (e.g., tablets, monitors, digital photo displayframes, computing displays, television displays, projected displays,etc.) using computer vision and augmented reality (AR) basedtechnologies in a visual and spatially intuitive manner. The embodimentsprovide processing for saving and retrieving virtual representations ofa cluster of displays and curate them locally or remotely using a mobiledevice (e.g., smart phone, tablet, wearable device, portable computingdevice, etc.).

One or more embodiments provide a unique way of visually identifying anddirectly interacting with a display or a cluster of displays, thatresults in intuitive interactions. Additionally, advanced userinteractions may be employed, such as drag and drop of selected contentdirectly on to visual representations of displays instead of textualrepresentations, such as network ID, BLUETOOTH® ID, etc. or of symbolicrepresentations such as folders or icons.

One or more embodiments provide spatially intelligent image editing andcuration of displays (where image means both still images—photo, andmoving images—video). Images may be cropped to aspect ratios, thatbetter match that of the displays that showcase them. Single images maybe processed to be spread across multiple displays in a spatiallycoherent manner. Selective display of certain important regions/featuresof an image may be displayed on selected displays. Special effects, suchas filters, may be assigned to specific displays through spatialinteractions.

One or more embodiments provide for saving and retrieving virtualrepresentations of a cluster of displays as virtual walls, which may beused to curate a cluster of displays even when geographically removedfrom the displays, while retaining the familiar user experience.Additionally, processing is provided to group clusters of displays intovarious groups and specify each group's properties such as collectivebehavior, content to be displayed, group special effects, etc. Dynamicreassignment of a particular display from one group to another isprovided and exceptions are available to group behavior.

One or more embodiments provide for easy planning, arrangement,alignment and mounting of a cluster of displays using computer visionand AR based technologies in a visual and spatially intuitive manner.Partial or full blending of displays with the environment is provided bymimicking their features such as color, texture, material and patterns.Processing provides for measurements, calculations and for suggestionsfor possible arrangements of displays along with specific measurementsfor placement of the displays. Immediate feedback about positioning andalignment of the displays is provided, and displays may be visuallypointed to for corrections, which guides users to get closer to thedesired configuration desired.

Based on the ambience of the display cluster, such as color of thephysical wall, available illumination, difference in color palettebetween the images in the display cluster, one or more embodimentsprovide for suggestions of suitable image filters that can enhancepresent images in the display in a visually pleasant manner. Thisambient information may be used to suggest photos and artwork, eitherfrom the local collection or from curated services that best match theambience of the room in which the display is placed.

One or more embodiments provide information for a far away or more idealviewpoint of the cluster of displays, even when standing near or totheir side by using perspective correction. The ambient information(such as color, texture and pattern of physical wall) is used to blendin with the environment, partially or fully by mimicking the details ofa wall in certain areas of the display. Emulation of various physicalmats of analog picture frames on digital picture frames is provided.

FIG. 1 is a schematic view of a communications system 10, in accordancewith one embodiment. Communications system 10 may include acommunications device that initiates an outgoing communicationsoperation (transmitting device 12) and a communications network 110,which transmitting device 12 may use to initiate and conductcommunications operations with other communications devices withincommunications network 110. For example, communications system 10 mayinclude a communication device that receives the communicationsoperation from the transmitting device 12 (receiving device 11).Although communications system 10 may include multiple transmittingdevices 12 and receiving devices 11, only one of each is shown in FIG. 1to simplify the drawing.

Any suitable circuitry, device, system or combination of these (e.g., awireless communications infrastructure including communications towersand telecommunications servers) operative to create a communicationsnetwork may be used to create communications network 110. Communicationsnetwork 110 may be capable of providing communications using anysuitable communications protocol. In some embodiments, communicationsnetwork 110 may support, for example, traditional telephone lines, cabletelevision, Wi-Fi (e.g., an IEEE 802.11 protocol), Bluetooth®, highfrequency systems (e.g., 900 MHz, 2.4 GHz, and 5.6 GHz communicationsystems), infrared, other relatively localized wireless communicationprotocol, or any combination thereof. In some embodiments, thecommunications network 110 may support protocols used by wireless andcellular phones and personal email devices (e.g., a Blackberry®). Suchprotocols may include, for example, GSM, GSM plus EDGE, CDMA, quadband,and other cellular protocols. In another example, a long rangecommunications protocol can include Wi-Fi and protocols for placing orreceiving calls using VOIP, LAN, WAN, or other TCP-IP basedcommunication protocols. The transmitting device 12 and receiving device11, when located within communications network 110, may communicate overa bidirectional communication path such as path 13, or over twounidirectional communication paths. Both the transmitting device 12 andreceiving device 11 may be capable of initiating a communicationsoperation and receiving an initiated communications operation.

The transmitting device 12 and receiving device 11 may include anysuitable device for sending and receiving communications operations. Forexample, the transmitting device 12 and receiving device 11 may includemobile telephone devices, television systems, cameras, camcorders, adevice with audio video capabilities, tablets, wearable devices, and anyother device capable of communicating wirelessly (with or without theaid of a wireless-enabling accessory system) or via wired pathways(e.g., using traditional telephone wires). The communications operationsmay include any suitable form of communications, including for example,voice communications (e.g., telephone calls), data communications (e.g.,e-mails, text messages, media messages), video communication, orcombinations of these (e.g., video conferences).

FIG. 2 shows a functional block diagram of an architecture system 100that may be used for configuring and controlling display devices 1-N 140(e.g., tablets, monitors, digital photo display frames, computingdisplays, television displays, projected displays, etc.; where N is ≥0)including content curation using an electronic device 120. Both thetransmitting device 12 and receiving device 11 may include some or allof the features of the electronics device 120. In one embodiment, theelectronic device 120 may comprise a display 121, a microphone 122, anaudio output 123, an input mechanism 124, communications circuitry 125,control circuitry 126, a camera 128, a visual app 129 (for controllingdisplay device content curation, management, arrangement, etc.), and anyother suitable components. In one embodiment, applications 1-N 127 areprovided and may be obtained from a cloud or server 130, acommunications network 110, etc., where N is a positive integer equal toor greater than 1.

In one embodiment, all of the applications employed by the audio output123, the display 121, input mechanism 124, communications circuitry 125,and the microphone 122 may be interconnected and managed by controlcircuitry 126. In one example, a handheld music player capable oftransmitting music to other tuning devices may be incorporated into theelectronics device 120.

In one embodiment, the audio output 123 may include any suitable audiocomponent for providing audio to the user of electronics device 120. Forexample, audio output 123 may include one or more speakers (e.g., monoor stereo speakers) built into the electronics device 120. In someembodiments, the audio output 123 may include an audio component that isremotely coupled to the electronics device 120. For example, the audiooutput 123 may include a headset, headphones, or earbuds that may becoupled to communications device with a wire (e.g., coupled toelectronics device 120 with a jack) or wirelessly (e.g., Bluetooth®headphones or a Bluetooth® headset).

In one embodiment, the display 121 may include any suitable screen orprojection system for providing a display visible to the user. Forexample, display 121 may include a screen (e.g., an LCD screen) that isincorporated in the electronics device 120. As another example, display121 may include a movable display or a projecting system for providing adisplay of content on a surface remote from electronics device 120(e.g., a video projector). Display 121 may be operative to displaycontent (e.g., information regarding communications operations orinformation regarding available media selections) under the direction ofcontrol circuitry 126.

In one embodiment, input mechanism 124 may be any suitable mechanism oruser interface for providing user inputs or instructions to electronicsdevice 120. Input mechanism 124 may take a variety of forms, such as abutton, keypad, dial, a click wheel, or a touch screen. The inputmechanism 124 may include a multi-touch screen.

In one embodiment, communications circuitry 125 may be any suitablecommunications circuitry operative to connect to a communicationsnetwork (e.g., communications network 110, FIG. 1) and to transmitcommunications operations and media from the electronics device 120 toother devices within the communications network. Communicationscircuitry 125 may be operative to interface with the communicationsnetwork using any suitable communications protocol such as, for example,Wi-Fi (e.g., an IEEE 802.11 protocol), Bluetooth®, high frequencysystems (e.g., 900 MHz, 2.4 GHz, and 5.6 GHz communication systems),infrared, GSM, GSM plus EDGE, CDMA, quadband, and other cellularprotocols, VOIP, TCP-IP, or any other suitable protocol.

In some embodiments, communications circuitry 125 may be operative tocreate a communications network using any suitable communicationsprotocol. For example, communications circuitry 125 may create ashort-range communications network using a short-range communicationsprotocol to connect to other communications devices. For example,communications circuitry 125 may be operative to create a localcommunications network using the Bluetooth® protocol to couple theelectronics device 120 with a Bluetooth® headset.

In one embodiment, control circuitry 126 may be operative to control theoperations and performance of the electronics device 120. Controlcircuitry 126 may include, for example, a processor, a bus (e.g., forsending instructions to the other components of the electronics device120), memory, storage, or any other suitable component for controllingthe operations of the electronics device 120. In some embodiments, aprocessor may drive the display and process inputs received from theuser interface. The memory and storage may include, for example, cache,Flash memory, ROM, and/or RAM/DRAM. In some embodiments, memory may bespecifically dedicated to storing firmware (e.g., for deviceapplications such as an operating system, user interface functions, andprocessor functions). In some embodiments, memory may be operative tostore information related to other devices with which the electronicsdevice 120 performs communications operations (e.g., saving contactinformation related to communications operations or storing informationrelated to different media types and media items selected by the user).

In one embodiment, the control circuitry 126 may be operative to performthe operations of one or more applications implemented on theelectronics device 120. Any suitable number or type of applications maybe implemented. Although the following discussion will enumeratedifferent applications, it will be understood that some or all of theapplications may be combined into one or more applications. For example,the electronics device 120 may include an automatic speech recognition(ASR) application, a dialog application, a map application, a mediaapplication (e.g., QuickTime, MobileMusic.app, or MobileVideo.app),social networking applications (e.g., Facebook®, Twitter®, Etc.), anInternet browsing application, etc. In some embodiments, the electronicsdevice 120 may include one or multiple applications operative to performcommunications operations. For example, the electronics device 120 mayinclude a messaging application, a mail application, a voicemailapplication, an instant messaging application (e.g., for chatting), avideoconferencing application, a fax application, or any other suitableapplication for performing any suitable communications operation.

In some embodiments, the electronics device 120 may include a microphone122. For example, electronics device 120 may include microphone 122 toallow the user to transmit audio (e.g., voice audio) for speech controland navigation of applications 1-N 127, during a communicationsoperation or as a means of establishing a communications operation or asan alternative to using a physical user interface. The microphone 122may be incorporated in the electronics device 120, or may be remotelycoupled to the electronics device 120. For example, the microphone 122may be incorporated in wired headphones, the microphone 122 may beincorporated in a wireless headset, the microphone 122 may beincorporated in a remote control device, etc.

In one embodiment, the camera module 128 comprises one or more cameradevices that include functionality for capturing still and video images,editing functionality, communication interoperability for sending,sharing, etc. photos/videos, etc.

In one embodiment, the visual app 129 comprises processes and/orprograms for managing content, curating content and providingarrangement of display devices. The content may include: visual content,graphic images, video content, photos, etc. The visual app 129 mayinclude any of the processing, curating, controlling, managing andarranging of content and/or display devices as described below.

In one embodiment, the electronics device 120 may include any othercomponent suitable for performing a communications operation. Forexample, the electronics device 120 may include a power supply, ports,or interfaces for coupling to a host device, a secondary input mechanism(e.g., an ON/OFF switch), or any other suitable component.

FIGS. 3A-D show an example of continuous lock of a virtual overlay 410on the electronic device 120 for a display device 140, according to anembodiment. In one example, as the user moves around holding theelectronic device 120, keeping the display device 140 within the fieldof view of the electronic device 120, the virtual overlay is continuallymatched and locked to the respective display. In one embodiment, this isachieved by means of AR technologies that combine pattern matching(e.g., pattern matching using instance-level object detection,category-level object recognition, block-based visual-pattern matching,etc.), computer vision and computer graphics. In one example, a patternmatching processor (e.g., processor 3111, FIG. 29) or process (e.g.,visual app 129 or content curation and display management processing3130), which is part of the AR library (e.g., part of visual app 129and/or applications 1-N 127), resides on electronic device 120 and istrained to detect a finite set of images. When the electronic device 120is pointed at a display device 140 and if the image being displayed inthe display device 140 is present in the electronic device 120's patternmatching database, the pattern matching process or processor providesthe AR library with details such as presence of an image, and the nameof the image identified. The AR library then uses its computer visionprocess or processor to determine where in the scene this pattern ispresent, and details such as the location of its four corners. The ARlibrary then uses its computer graphics process or processor to overlaydesired visual effects within the detected four corners, that providethe illusion of a virtual overlay that remains locked to display device140.

In one example, the virtual overlay 410 may allow advanced userinterfaces elements to be superimposed on top of them. These elementsmay include buttons and widgets that may be used to modify the state ofthe display device 140 (such as color, saturation, filters, etc.) towhich the electronic device 120 is pointed at. Additionally, moreadvanced interactions such as dragging content/effects from a gallery onto the virtual overlay 410 are also possible. Putting all the abovetogether, a user may detect and manipulate properties of a displaydevice 140 simply by pointing at it with the electronic device 120. Thisallows for very intuitive visual and spatially coherent interactionswith display devices 140 and provides a satisfying user experience.

The above mentioned computer vision technology also provides thecapability to measure the position, orientation, aspect ratio anddimensions of a display device 140 with respect to its neighbors. In oneembodiment, the AR library simultaneously tracks multiple image targetsusing its pattern matching library (e.g., part of visual app 129 and/orapplications 1-N 127). This allows the electronic device 120 to setupthese virtual overlays on top of each display device 140 in a cluster ofdisplays which is displaying a familiar image. This collection ofvirtual overlays may then be analyzed spatially to determine whichdisplay device 140 is to the left of the other display devices 140,above yet another display device 140, etc. The relative positioning of acluster of display devices 140 can thus be determined. Based on therelative orientations of the edges of one virtual overlay with respectto that of another, the electronic device 120 determines the relativeorientations of these display devices 140. The relative orientation of acluster of display devices 140 can thus be determined. It should benoted that the aspect ratio of a display device 140 is the ratio betweenthe width and height. Display devices 140 may have formatting in varyingpopular aspect ratios, such as 4:3, 16:9, 16:10 etc., depending on theiruse case and the appropriate media with the correct aspect ratio needsto be displayed in the display device 140 in order to avoid letterboxingor presence of black lines on the horizontal or vertical extremes. Eachdisplay device 140 inherently knows its own aspect ratio and physicaldimensions. As soon as the electronic device 120 is pointed at displaydevice 140 and identified, the electronic device 120 uses networkcommunication to query display device 140's aspect ratio and dimensions,which is communicated by display device 140 and used by electronicdevice 120 in further calculations. In one embodiment, the aspect ratioof display device 140 may be determined without network communication.With the virtual overlays mentioned above, electronic device 120 maycompare the width and height of the virtual overlays to determine theobserved aspect ratio of the display device 140. When presented with acluster of display devices 140, this technology may be used to assessthe relative placement, alignment and size differences of one displaydevice 140 with respect to another display device 140 within thecluster. Since the above mentioned computer vision technology may beused to assess the placement, alignment and size of one display device140 with respect to another display device 140, it may be used toeffectively spread a single image across multiple display devices 140 ina spatially coherent manner, that takes into account the gap betweenthem (see, e.g., FIGS. 7D and 8D).

In one embodiment, the multi-display device 140 environment illustratedin FIG. 5A provides identification of physical aspects of the displaydevices 140 may be performed using the mobile device 120, according toan embodiment. In one example, a single display device 140 or a clusterof display devices 140, are also referred to as modular display devices140 of the same or varying sizes. The display devices 140 are allconnected on a same network. The network may be any wired or wirelesscommunication channel such as Wi-Fi, BLUETOOTH®, LAN, WAN, cellular,etc. Each display device 140 has a unique network ID that may be used toassess and communicate with other devices connected to the network. Eachdisplay device 140 may or may not have touch or touchless gesturalinteraction features. In one example, an initial setup process may berequired to add a display device 140 into a local network. For example,a user may pass local network configuration data from an electronicdevice 120 to each display device 140 through other technologies it maypossess such as near field communication (NFC), USB, etc.

In one embodiment, the display devices 140 may include a spectrum oflight emitting diodes (LEDs) (visual or infrared) or other physicalmarkers on the bezel/border of the display device 140 frames, allowingdistinct identification of each display device 140 using computer visiontechnology. In one example, a user has access to a mobile device 120with a camera 128, with input technologies such as multi-touch or airgesture, which is on the same network as the rest of the display devices140. The camera's 128 field of view may be adjustable to view a widearea or zoomed in to focus on a narrow area. In one example, the displaydevices 140 may also be connected via the network to other entertainmentdevices such as TVs and speakers.

In one embodiment, the electronic device 120 performs interactions withand commands a display device 140 by using the camera 128 to view thedisplay device 140 and use of the visual app 129. This interaction maybe referred to as direct interaction/manipulation. In one example,tracking of display devices 140 is provided using the electronic device120, which is the ability to determine the position, orientation andsize of each display device 140 with respect to other display devices140, the electronic device 120 and the room the display devices 140 arelocated.

Computer vision has progressed to a point where several processes may beused to track display devices 140.

-   -   Frame Markers: Display devices 140 may display specific        predefined patterns known as markers, with which they may be        identified. These markers typically take the form of        checkerboard like patterns, that are unique. There are several        types of these markers in varying complexities. In one example,        QR codes and bar codes are specific examples of markers.    -   Image targets: Advances in pattern matching provide use of        regular images as ways to identify and track a display device        140. This may be applied to either the currently displayed        image, or each display device 140 may have a unique default        image that it uses for the same purpose.    -   Character recognition: a display device 140 can display specific        text that may be used both for tracking and for visually        communicating other meta-data.    -   Watermarked markers: images displayed may be encrypted with        invisible watermarks (which can only be picked up by the camera        128) that may be used as a way to identify a display device 140.    -   LEDs embedded in the frame: visible/invisible LEDs or other        light emitting elements may be embedded in the bezel of a        display device 140. These elements may then be sequenced to        blink in a particular way and/or in specific colors, that will        allow identifying and tracking the display device 140.    -   Physical features of frame: with modern AR software being able        to pattern match 3D models and track in real time, display        devices 140 with unique physical features may serve as a way for        identifying and tracking a display device 140.

One or more embodiments are described below with image targetingtechnology. However, it should be noted that other tracking techniquesmay also be used as well.

Communication link technology described herein uses computer visiontechnology at its core, which allows identifying a known image whenviewed through a camera 128 of electronic device 120. In one embodiment,all display devices 140 in the network can constantly communicate witheach other to update each other with details such as name of thecurrently displayed image, their network ID, their aspect ratio, etc. Asa result, every display device 140 on the network, including theelectronic device 120, contains a look up table that shows therelationship between network ID and currently displayed image. Anexample for a look up table provided below:

Network Current ID Image 1 IMG3 2 IMG1 3 IMG5

In one embodiment, when an electronic device 120 is pointed to a clusterof display devices 140, it does not immediately know which displaycorresponds to which network ID. In one embodiment, the electronicdevice 120 has a pattern library, that constitutes its ‘known’ set ofimagery. Each pattern is associated with an image identifier. Theelectronic device 120 used the visual app 129 to constantly scan eachframe it captures with the camera 128 to see if any of the features inthe captured image match with any content of the pattern library. Whenthe electronic device 120 observes and recognizes a known image on adisplay 120, it then uses the look up table to identify the network IDof that specific display device 140 and henceforth uses it tocommunicate any subsequent commands or changes to that display device140. The pattern library used by the electronic device 120 may reside onthe electronic device 120 or be accessed on the cloud/server 130 (FIG.2). Every image captured via the camera 128 is put through apre-processing stage, where the image is analyzed, defined as a patternand added to the pattern library and to the cloud/server 130. Thecloud/server version of the pattern library may constantly synchronizewith other users and other devices to ensure it is up to date with data.In one embodiment, all updates to the pattern library is done directlyat the cloud/server. In another embodiment, device 120 or 140 initiatespattern library synchronization with the cloud/server every time itmakes a change to its local pattern library. Each display device 140 mayalso be pre-loaded with a unique image, which serves as its defaultimage and which in turn allows uniquely identifying the display device140.

In one embodiment, computer vision technology is also used to identifythe physical dimensions of the display device 140 and its relativeposition and orientation with respect to the display 121 of theelectronic device 120. Computer graphics technologies may be employed torender digital/virtual overlays 510 (FIG. 4B) on the display 121 of theelectronic device 120, that are on top of and encompass views of thesedisplay devices 140.

FIG. 4A shows an example of a cluster of multiple display devices 140,according to an embodiment. The relationship between the cluster ofdisplay devices 140 and their virtual overlays (e.g., overlays 510, FIG.4B) may be captured into a snapshot referred to as a virtual wall, whichis a combination of live imagery that shows the room/environment inwhich these display devices 140 are placed in (plants, desk, books,etc.) as well as their virtual overlays. One or more embodiments providesaving and retrieving of these virtual walls in order to manipulate themat any time. These virtual walls may be thought of as a photo of theroom, captured by the camera 128, where the content in the displaydevices 140 alone get updated with time. In one embodiment, the virtualwalls may take the form of a standard rectangular image. In anotherembodiment, these virtual walls may be a panoramic image, such as a360-degree (or other angle) image captured by either multiple cameras128 on the electronic device 120 or sweeping around the electronicdevice 120 with a single camera 128 or other known methods. As thephysical display devices 140 get updated with new images, the virtualwalls containing these display devices 140 have their correspondingvirtual overlay updated. Users may drag and drop content on to thesevirtual overlays within these virtual walls to update a real displaydevice 140, even when they are nowhere near the actual display device140. This allows for remote curation of a cluster of display devices140.

In one embodiment, once the user sets up a frame that satisfactorilycovers all interested display devices 140 within its Field of View (FOV)and tracking has been initiated to draw the above mentioned overlays,several types of information can be obtained with this tracking data:

-   -   a) The dimensions and aspect ratio of each display device 140;    -   b) The current orientation of each display device 140 (portrait        or landscape);    -   c) The relative sizes of display devices 140 with respect to        each other;    -   d) The position and orientation of display devices 140 with        respect to each other (in other words, the gap between them);    -   e) The distance of the electronic device 120 and hence the user        from the each display device 140 and the plane/wall on which the        display devices 140 are mounted; and    -   f) The relative orientation of the user with respect to each        display device 140 and the plane/wall on which the display        devices 140 are mounted.        This rich list of information listed above provides the visual        app 129 to add interesting capabilities and functionalities to        display device 140 control, configuration and content curation.

In addition to tracking the display devices 140, in one embodiment usersmay also supply ambient information through any of the followingoptions:

-   -   Stepping back or widening the FOV to include more of the ambient        information in the room;    -   Perform a separate pass, where the user captures just the        ambience, taking a photo of the part of the room or object that        they would like to be representative; and    -   Perform a separate pass, where the user performs a        movement/rotation to capture a stitched 360-degree image of the        room, to be used for ambience calculations.

As seen in FIG. 4A, three display devices 140 are hung on the wall. Inthis example, the three display devices 140 are in different shapes andsizes. It should be noted, however, that display devices 140 may assumethe same shape or size, take various shapes or sizes, or additionaldisplay devices 140 or less display devices 140 may exist to formspecific layout combinations. Each display device 140 displays content,which is either the factory default if the display device 140 has neverbeen configured, or a user supplied content recorded by the patternlibrary through previous content management of the display device 140.

In one embodiment, with the ambience information captured, it ispossible to assess certain details of the ambience, such as color of thewalls, lighting in the room, the color spectrum of all of the imagesshown in neighboring display devices 140, etc. In one embodiment, theapplication running on electronic device 120 switches betweentwo-dimension (2D) and AR mode. In one example, 2D mode allows forregular 2D interactions necessary for operations such as browsing imagegalleries and typing, etc. The AR mode loads the AR library and providesa camera view that is used to look at and interact with display devices140. In one embodiment, when the AR mode is turned on the electronicdevice 120, the display devices 140 switch to a content that isconducive to one of the image based tracking technologies describedabove. In one embodiment, the displayed content may take the form of apre-determined frame marker, watermarking etc., that is unique to thatdisplay device 140. In another embodiment, the currently displayed imageitself might serve as an image target and for tracking. The electronicdevice 120 is constantly communicated with the state of all displaydevices 140 and is correspondingly loaded with the right patternrecognition dataset.

FIG. 4B shows an electronic device 120 capturing a layout of the clusterof multiple display devices 140 of FIG. 4A, according to an embodiment.As shown, the electronic device 120 captures the layout of the displaydevices 140 of FIG. 4A and provides an overlay 510 over each of thecaptured displays 410. As the user points the camera 128 to the displays410, the electronic device 120 starts the pattern matching processdescribed above to recognize each one of the display devices 140.

In one embodiment, instead of solely relying on content recognition, theelectronic device 120 is able to distinguish one display device 140 fromanother based on the LEDs embedded around the frame, edge or border ofthe display device 140. For example, each display device 140 may have aunique LED lighting pattern. If there are X number of LEDs for eachdisplay device 140, the unique pattern can reach X power of 2. Foranother example, the LEDs may increase the unique pattern by controllingthe on/off frequency, on/off sequence, etc. In one embodiment, theelectronic device 120 only needs to distinguish one display device 140from another within the same group of captured display devices 140. Thesame pattern can occur for another display device 140 as long as itbelongs to a different group. This allows a smaller number of requiredunique patterns. The unique pattern may only need to present itselfduring a pairing process. In one embodiment, the pairing processincludes loading the credentials for network communication such as Wi-FiID, BLUETOOTH® network etc., and its respective password onto electronicdevice 120 and transferring the same to a display device 140 viatechnologies such as NFC. Once display device 140 receives thesecredentials, it is then live on the wireless network, enabling wirelesscommunication between electronic device 120 and display device 140henceforth. Once the display device 140 is paired with the electronicdevice 120, the identification of the display device 140 is associatedwith the corresponding virtual overlay as described above.

In one embodiment, when multiple display devices 140 are recognized, thevisual app 129 identifies the region or pixels of each of the recognizeddisplay devices 140 in the image captured by the electronic device 120.In one example, a box region (overlay 510) is created for eachrecognized display device 140 corresponding to the identified region orpixels, and represents the corresponding virtual object. These boxregions remain associated with the corresponding display device 140, andeach maintains the aspect ratio most suited for the display capabilityof the corresponding display device 140.

FIGS. 5A-F show a sequence diagram for an example of managing the imagecontent displayed on the cluster of multiple display devices 140,according to an embodiment. In FIGS. 5A-F, three displays are shown,display 640, 641 and 642, all which are similar to display device 140 asdescribed above. In FIGS. 5A-5D, image content 651 is displayed indisplay 641. In one example, the user can use the same control means asdescribed above to upload content into desired display 641. For example,by dragging a picture from an on-device album or gallery 630 to one ofthe box regions shown on the electronic device 120. FIG. 5D showsdragging 661 from the album or gallery 630 to the box region associatedwith display 641. In FIG. 5E the image dragged from the album or gallery630 is now displayed on display 641 as image content 652, which replacedimage content 651. FIG. 5F shows the resulting wall of displays 640, 641and 642.

FIGS. 6A-D show a sequence diagram of an example for managing the imagecontent 711 displayed on a display device 701 by adjusting scale,according to an embodiment. Users today have a large number of images intheir collection. These images (e.g., photos, videos, etc.) are oftentaken with varying devices such as digital cameras, smartphones etc. Inaddition, they also receive/have access to images from various socialplatforms and services such as FACEBOOK®, INSTAGRAM®, PINTEREST®,SNAPCHAT®, etc. It would be expected that the user would like to shareany of these images on their display 701 (similar to display device 140as described above). Unfortunately, these images can come in variousdimensions and aspect ratios and might not match 1:1 with the aspectratio of the intended display 701.

Auto cropping mechanisms such as center cropping the image on droppingto fit the display's 701 aspect ratio do not provide satisfactoryresults as region of interest in the image could be anywhere within theimage and center cropping could cut out information or present them inan unflattering manner. Manually having to figure out the aspect ratioof the image, go into a third party app and perform cropping is tedious.Since the modular displays are always networked and constantlycommunicate with each other the details such as physical details andstate, it is possible to know when the image being dropped on to amodular display will not match. In one embodiment, as shown in FIG. 6Aon release of dragging an image content 720 from the album or gallery630, a new pop up 730 appears as shown in FIG. 6B. The new pop up 730shows the dragged image 720 along with a rectangle in the aspect ratio(the ratio of the width to the height) of the display that the user hasdragged to. The user can then move and scale the box (with aspect ratiolocked) until it best fits the regions the user would like to display.On confirming, this subset of the image alone will show up in thedisplay 701 as the adjusted image content 721 as shown in FIG. 7D.

In another embodiment, upon drag and drop and identifying a mismatch,the visual app 129 may also suggest another display device 140 in agroup of displays that would be a better fit for this particular aspectratio, so the user would not have to crop and compromise the look andfeel of the image. This recommendation to use another display device 140may also be based on other parameters such as colors of neighboringdisplay devices 140, motion of the content, resolution and otherfeatures. For example, consider a scenario where a cluster of displaydevices 140 are arranged on an environment (e.g., a wall), such that thedisplay devices 140 on the left side show images that are predominantlyon the red color spectrum and the display devices 140 on the right sideshow images that are predominantly on the blue color spectrum. An imageis dragged and dropped that is predominantly orange in color, on to adisplay device 140 on the right side, the orange would appear glaringlyout of place among the display devices 140 with predominantly blueimages. One or more embodiments perform an analysis and suggest that theimage might best be placed in one of the display devices 140 on the leftfor a visually pleasing effect. The user may then choose to select therecommendation or ignore it if they intended to have such a contrastinglook (e.g., for creative reasons).

One or more embodiments provide for remotely managing the image contentdisplayed on a display device (e.g., display device 140), according toan embodiment. In one embodiment, a wall may include several displaydevices 140. In one embodiment, when a user captures an image of thedisplay devices on the wall with an electronic device 120, the imagealso captures the contextual information surrounding the displaydevices, such as other wall decorations (e.g., surrounding furniture,color contrast among the fixtures, plants or other environment settings,etc.), which is then saved as a virtual wall. The contextual informationavailable in these virtual wall images may improve the flexibility ofuser interaction with the modular devices. For example, the user maystay in her bedroom to manage the content of the modular devices in theliving room. The virtual wall, which is a visual snapshot of a clusterof display devices 140, can also be thought of as a list of networkaddresses such as an IP address embedded spatially within the snapshot.With the network address in hand, the respective display device 140 canbe controlled from anywhere within the home. If the internal network isconnected to external networks such as the Internet, the user has theability to curate these display devices 140 from remote locations.Alternatively, another user that has access to the network address ofthe display devices 140 but is located remotely, can also curate thedisplay devices 140.

The user may use one or more virtual walls to manage the content displaychanges when she is on vacation, for example in the Caribbean (thisexample may require a home hub that can interact with the electronicdevice 120 so that the electronic device 120 can identify the specificmodular display device to the hub, and through the hub to complete thecontent management). Alternatively, this can also be achieved via thecloud, if the multiple display devices are connected to the internet viatechnologies such as Wi-Fi.x1. The context information in these virtualwalls allows her to not only view the content that will be displayed oneach individual modular device, but also the overall effect of thecontent display in view of the surrounding setting. In this example, theuser is able to save these virtual walls and categorize them withrepresentative names such as hallway, guest room, living room, etc.

In addition to the above, users can share their virtual walls withfriends, so they can briefly allow their friend to curate their photowall. Alternatively, users can dedicate specific displays to certainfriends, allowing them to curate that display. An example would be amother dedicating a display to her daughter who is away at college. Thedaughter has access to this display and constantly updates it with herfavorite images, to inform her mother of her life activities. This canprovide for some fun and intimate interactions.

In one example, the image content is selected from an album or gallery.The user is able to drag and drop selected image content onto an overlayassociated with a selected display device located on the wall. Uponrelease of the image content, the image content is now displayed on theselected display device.

FIGS. 7A-D show diagrams of an example for managing the image contentdisplayed on multiple display devices by spreading a single contentacross the multiple displays 901, 902 and 903 (similar to display device140 as described above) in a spatially coherent manner, according to anembodiment. In one example, image content is spread across the displays901, 902 and 903 in a spatially coherent manner. FIG. 7A shows acaptured image of a layout of the displays 901, 902 and 903 on anelectronic device 120 with an album or gallery 630 available for contentmanagement and control. The displays 901, 902 and 903 each show imagecontent 911, 912 and 913, respectively. To spread an image across thedisplays, the user needs to perform 3 steps: a) first a menu option isselected, expressing intent to enter into this mode; b) the virtualoverlays of displays 901, 902, 903 are then tapped to let the systemknow that it is across these chosen displays that it is desired tospread an image; c) an image may then be dragged and dropped onto thevirtual overlays of any one of the selected displays. Each display isthen sent a message to load the image of interest along with the texturecoordinates valid for that particular display, which the display uses toensure it displays only the portions of the image that it is supposedto. Texture coordinates are pairs of values used to indicate a specificsub-region of an image. The texture coordinates may be used tocommunicate the exact four points on an image of a display device 901,902 and 903, the pixels within which a display device needs to render.Depending on the relative positions of each of the displays 901, 902 and903 with respect to each other and with respect to the electronic device120, their aspect ratios and physical dimensions, each display 901, 902and 903 will be calculated to subtend a particular sub-region of theimage to be displayed and thereby the corresponding texture coordinates.The physical analogy would be to take a large print of a photo and thenoverlay it on top of a few picture frames and figure out which regionsof the photo actually fall on top of which picture frames, cut them out,drop them into the respective picture frames, and discard the rest ofthe photo.

In FIG. 7B, a user selects an image from the album or gallery 630 thatshe wants to be displayed across the displays 901, 902 and 903simultaneously, and drags 930 the selected image and drops it onto oneof the displays 901, 902 and 903 as shown in FIG. 7C. In FIG. 7D thedropped image gets spread across the three displays 901, 902 and 903 asimage content 921, 922 and 923. This technique enables a solution tomaintain the spatial relationship of the image content over the displays901, 902 and 903. It may be observed that the slope of the hill in theimage content 921, 922 and 923 is maintained. This would not have beenthe case if spatial relationship of frames had not been taken intoconsideration.

FIGS. 8A-D show diagrams of an example for managing the image contentdisplayed on multiple display devices by overlaying on top of a selectedimage content during the content selection process so that the imagecontent is cropped over a whole group 1000 of display devices includingthe spaces among them, according to an embodiment. FIGS. 8A-D provide anextended example of the concept shown in FIGS. 7A-D with nine displays.In this example, instead of dividing the selected image content 1030into nine parts according to the shape and size proportionally, FIG. 8Cillustrates that the display devices overlay on top of the selectedpicture during the content selection process so that the image contentis cropped over the whole group 1000 of the display devices, includingthe spaces among them. The user, thus, may understand how the imagecontent will look like when it is displayed across the devices in thegroup 1000. Note that each box region in the group of box regions 1040of FIG. 8C contains a part of the image content 1030. Moreover, all ofthe box regions altogether cover only a partial image of the selectedimage content 1030. FIG. 8D illustrates how the selected image contentis displayed by the multiple display devices with the display device1001 showing its cropped image content 1041. Note that the displaydevices display only the parts of the image that were overlapped in thecorresponding box regions in FIG. 8C.

In one embodiment, a user has the option to more flexibly adjust how theselected content will be displayed on each of the display devices. Forexample, via gestures such as slide (move up, move down, move left andmove right), rotate, scale and crop, the selected image content 1030changes its position, orientation, scale and boundaries relative to thegroup 1040 of the box regions (the relative positions of the box regionsare fixed with respect to each other). In one example, the group 1040 ofbox regions remains still as the user manipulates the selected imagecontent 1030 around those fixed box regions. The selected image content1030 then populates into those fixed box regions as desired. This allowsthe user to fit the most interesting content in the best way possible,to their liking.

FIGS. 9A-F show diagrams of an example for managing the image contentdisplayed on display devices 1101, 1102 and 1103 by spreading a singlecontent across these multiple display devices based on regions ofinterest, according to an embodiment. In one example, a technique forspreading a single content across multiple display devices 1101, 1102and 1103 is based on the box regions (of interest) 1131, 1132 and 1133.FIG. 9A shows the display devices 1101, 1102 and 1103 secured onto thewall and a captured image of them on an electronic device 120. Thedisplay devices 1101, 1102 and 1103 are currently showing image content1111, 1112 and 1113, respectively.

The user selects image content from an album or gallery 630 that shewants to be displayed across these display devices 1101, 1102 and 1103simultaneously and drags and drops it into any one of those displaysshown on the electronic device 120. FIG. 9B illustrates an initial viewof how the selected image will be displayed across the display devices1101, 1102 and 1103 as image content 1121, 1122 and 1123, based on theoriginal box regions 1131, 1132 and 1133. As illustrated by FIGS. 9B-F,only the part of the selected image that falls within a box region willbe displayed.

In FIG. 9B, the body of the people in the selected image content are notcorrectly/fully included in any of the box regions 1131, 1132 and 1133.One or more embodiments provide the ability to upload the most desiredportion of a selected image content into any of the display devices.Each box region is designated to a particular display and representswhat pixels are drawn on that particular display. The aspect ratio of abox region is always locked to that of the display device it is assignedto (even when its scale changes, it maintains the aspect ratio). FIG. 9Cshows that a user is able to select any of the box regions 1131, 1132and 1133 and drag one of its corners 1140 to change the area that thisbox region covers.

FIGS. 9D-E show how a box region can be moved so that it overlays thedesired portion of the selected image content. As described above, a boxregion on the captured image is associated with a particular displaydevice. When the box region has its corners 1141 dragged, moved orscaled, the process tracks the changes and records the new position ofthe box region without losing the display device association. FIG. 9Fshows the final result of such operation that each display device 1101,1102 and 1103 is able to display the exact portions 1121, 1122 and 1123of the selected image content, as designated by the user via theelectronic device 120.

FIGS. 10A-D show diagrams of an example for managing the image contentdisplayed on display devices by changing the relative relationship ofthe image content 1211 displayed on a display device 1201, according toone embodiment. In one example, the visual app 129 provides the abilityfor a user to change the relative relationship of the image contentdisplayed on the display devices. In one embodiment, the user is able tomove a box region corresponding to a display device on the right side tooverlay on any part of the selected image 1240 (selected from the albumor gallery 1230 in FIG. 10B). For example, the display device 1201 onthe right may display an image of the kid on the right side of theselected image 1240. The user is also able to scale the size of the boxregion 1250 to adjust the portion of the image content that will beactually displayed on the corresponding display device 1201. This allowsthe user to change the ratio of different aspects of the image contentdisplayed by the display devices, with respect to the overall selectedimage 1240. In another embodiment, a small inset image is shown thatshowcases the original alignment of each box with respect to the imagecontent that serves as a reference for the user. Once the box region1250 is dragged to the portion of the selected image content 1240 asdesired (and adjusted for scaling), the final image content 1221 isdisplayed on the display device 1201 in FIG. 10D.

FIGS. 11A-F show a sequence diagram of an example for sharing imagecontent shown in any of the multiple display devices, according to oneembodiment. FIG. 11A shows eight display devices (similar to displaydevice 140). In FIG. 11B a captured image of the display devices(including display device 1301 showing image content 1311) is shown onthe electronic device 120, along with an album or gallery 1330.Similarly as a user may drag and drop an image content from theelectronic device 120 on to a display device with an overlay, they maydo the opposite as well in order to save or share the image content. Inone example, a user may point their electronic device 120 to a displaydevice and obtain access to the original image content (e.g., imagecontent 1311 on display device 1301) displayed in the display device forpurposes such as saving, sharing and swapping.

Typically, if a user would like to share an image/video that iscurrently displayed in one of the display devices with a friend, theywould need to navigate through the plethora (e.g., 100s, 1000s) ofphotos in their mobile device to search, identify, attach to a messageor email and share it with another user. In one embodiment, using an ARtechnique, the visual app 129 identifies the photo/video, links directlyto the local copy stored in its gallery or album and makes it availableimmediately for further operations, such as sharing with a friend,sharing with another mobile app, etc. The AR tracking here worksanalogous to shortcuts on desktop computers.

The network look up table described above along with virtual overlays isuseful in identifying a display based on its currently displayed image.The same look up table may also be used to obtain the original imagethat a particular display device 140 is currently displaying. When theuser directly touches a virtual overlay associated with a display device140, the system then uses the network address associated with thedisplay on the look up table to retrieve the path to the image that thedisplay is currently showing. Once this path is retrieved, the systemcan proceed to do any number of operations with it. In FIG. 11B the boxregion 1320 is selected by a user. In FIG. 11C the selected imagecontent 1311 in the box region 1320 is dragged to the share portion 1340of the display on the electronic device 120 as shown in FIG. 11D. InFIG. 11E, a selection graphic 1350 (e.g., a window, a pop-up, etc.) isdisplayed with the selected image content 1311 providing options forsharing, forwarding, saving, etc. (e.g., via text message, email,messenger, saving to a gallery or album, social media platform, etc.).In one example, the user selects to share the image content 1311 in atext message, and FIG. 11F shows a text message display 1360 with theimage content 1311 that may be forwarded to others as desired. Inanother example, the user may select the image content 1311 in the boxregion 1320 and drag the image content 1311 to a portion of the displayon the electronic device 120 such as the gallery or album, a favoritesselection, etc. for saving the image content 1311 as desired.

FIGS. 12A-B show an example illustration of finger movement drag anddrop for moving image content between display devices 1401, 1402 and1403, according to one embodiment. In one example, display devices 1401,1402 and 1403 are displaying image content 1411, 1412 and 1413,respectively. In this example, a user desires to move the image content1412 in the display device 1402 to the display device 1401. The user maygrab the image content in the box region 1440 shown on the electronicdevice 120 by dragging and dropping it on to the desired display, inthis example the box region associated with display device 1401. Thisway, the user may swap and reorganize the same set of images among thecluster of display devices, without having to reach into their galleryor album 1430. The result is shown in FIG. 12B where the content image1412 is shown in display device 1401. In one example, another imagecontent 1422 takes the place of the moved image content 1412. The imagecontent 1422 may be a default image, a next image in the gallery oralbum 1430, a previously displayed image content (e.g., the lastdisplayed image content or a shuffle selection) recorded by the displaydevice, etc. In another embodiment, the content 1422 might be the sameas 1412 i.e., a simple swap of images happens between displays 1401 and1402.

FIGS. 13A-B show an example illustration of finger movement drag anddrop for applying predetermined filters or filters based on ambientcolors for display devices, according to one embodiment. In addition todragging and dropping image content on to a display device, the sametechnique may be used to apply effects on a particular display device.FIG. 13A shows an example, where the user points the electronic device120 running the visual app 129 at the display device 1501 that isdisplaying image content 1511. Once the display device 1501 isrecognized, the user can select an option from a list of image effects1530 such as ‘black and white’, ‘sepia’, ‘vignette’, etc. and drop themon to the display device show in on the electronic device 120. Thecommand is conveyed via the network and the intended display deviceapplies the selected effect 1531 as updated image content 1521 shown ondisplay device 1501 in FIG. 13B.

In one embodiment, the electronic device 120 also detects the color ofthe wall that the display device(s) (e.g., display device 1501) is(are)hung/attached to and the lighting color in the room—referred to asambient colors. Considering the color scheme of the content image(s)showcased in the display device(s), and the color scheme in theambience, the visual app 129 provides a recommendation for colorfilters, that can neutralize the overall color palette of the collectionand allows them to blend into the room in a more visually pleasingmanner.

FIGS. 14A-C show an example illustration of selecting colors from theimage content 1611 being displayed to be used as the mat color fordisplay devices (e.g., display device 1601), according to oneembodiment. Picture frames available on the market typically areprovided with a mat. Mats usually serve the purpose of bringingattention to the displayed image content by providing a color relief andseparation from the plethora of information surrounding the frame. Thesemats are mostly made of paper, but do also come in a variety of texturesand colors. FIG. 14A shows a display device 1601 having a mat portion1620 and image content 1611. In one example, it is desired to modify themat portion 1620. In FIG. 14B the captured image content is shown on theelectronic device 120. A graphic or icon 1630 is used to select aportion of the image content shown on the electronic device 120. In FIG.14C the graphic or icon 1630 is dragged to a portion of the imagecontent and a color representation (e.g., a graphic display of the colorselected) 1640 of the color of the portion of the image content wherethe graphic or icon 1630 is placed is displayed. The image mat 1620 isthen changed to mat 1621 to match the selected representation 1640 anddisplayed on the display device 1601.

FIG. 15 shows an example illustration of grouping of display devices inmultiple areas, according to one embodiment. The home of today may befilled with display devices (e.g., display devices 140) of varyingshapes and sizes—television sets, digital picture frames, tablets, etc.These display devices are also typically set in different parts of thehome. Especially when it comes to digital picture frames, people mayprefer to decorate different regions of the home with images ofdifferent themes, albums, etc. In some cases, people also break down acluster of picture frames into two or more themes. For example, imagecontent on the wall 1700 in one room (e.g., a living room) in displaydevices 1701 and 1702, and image content on the right on wall 1710 indisplay devices 1703, 1704 and 705 in another room (e.g., a kitchen).Other scenarios such as certain image content on the left, center andright of a wall may have separate themes of image content. For example,boy's pictures on the left of a wall, family or parent pictures in themiddle of the wall, and girl's pictures on the right of a wall. In suchsituations, there arises a need to group a set of display device.

In one embodiment, grouping display devices provides the followingbenefits:

-   -   Provides interacting/modifying with a group of display devices        without disturbing another neighboring (or far away) group;    -   Provides setting characteristics that apply to all display        devices in a group—such as group effects/filters, slide show        animation speed, etc.; and    -   Provides curation of all display devices in a group with one        quick interaction.

In the example shown in FIG. 15, the display devices 1701 and 1702 areset up as a group 1711 (e.g., a color or other indication may beassociated with the group for easy visual recognition when captured onan electronic device 120 (FIG. 2)), and the display devices 1703, 1704and 1705 are set up as a group 1712 (which would be another color orother indication to provide recognition). This clean distinctionprevents any interaction on the living room (wall 1700) images toadversely affect the set of display devices in the kitchen (wall 1710)and vice versa.

FIGS. 16A-C show an example illustration of managing groups of displaydevices using an electronic device 120, according to one embodiment.There are several interaction opportunities for both the creation ofdisplay device groups and for modifying/curating them. In one example,users may use interactions to create the desired groups. FIG. 16A showsan example where a tray of indicators (e.g., colors) 1810 and 1815 showsup in the electronic device 120. The display devices 1801, 1802, 1803,1804, 1805 and 1806 are desired to be placed into different groupings of1810 and 1815. In one example, the group 1810 includes display devices1801, 1802, 1803 and 1804. The group 1815 includes display devices 1805and 1806. Users can select the grouping selection from selections 1830,point their electronic device 120 at a display device and drag and dropan indicator (e.g., a color) on to the represented or captured displaydevice. On the electronic device 120, indicators (e.g., colors) for thetwo groups (1810 and 1815) are shown on the bottom portion of theelectronic device 120, along with a selection for adding more groups.

In FIG. 16B a selection is made for changing display device 1801 fromgroup 1810 to group 1815 using drag and dropping of the group 1815indicator onto the representative image 1850 for display device 1801.FIG. 16C shows completion of the assignment of display device 1801 togroup 1815. Upon completion, the display device 1801 is now assigned togroup 1815 (e.g., an orange group). All displays assigned with the group1815 indicator (e.g., orange color) will become part of the same group.Similarly, displays assigned with a group 1810 indicator (e.g., bluecolor) will all become part of the blue group. Users can select the plus(+) button on the left to create more such groups. Users may dynamicallyarrange and rearrange groups as they please. Any display device can atany time be made to:

-   -   Join a group;    -   Leave a group and join another group; and    -   Become its own island group, with just that display device        within that group.

Group curation actions may in general be classified into four subsets:One-to-one; One-to-many; Many-to-many; and Many-to-one. The prefixdenotes the nature of source selection on the electronic device 120, andthe suffix denotes the nature of destination assignment on drop. Thebehavior and output of actions change based on which of the above modesthat the user chooses.

One-to-one refers to the technique where a user may perform one actionto influence one display device. For example, the drag and droptechnique is used to apply a single image content to a single displaydevice as described above is an example of a one-to-one interaction.Similarly, dragging and dropping a filter to apply to a single displaywould be one-to-one.

One-to-many refers to a technique used when a set of display devices aregrouped. Users may perform familiar drag and drop actions to influenceone action on all the display devices in a group. For example, to applya filter effect to every display device in a group, a user may drag anddrop the filter on to any display device in the group and the rest ofthe display devices in the same group will also be applied with the samefilter.

Many-to-many refers to the technique when instead of selecting a singleimage content from an album and dragging it, if the user drags the albumicon and drops it on to any display device in the group, all displaydevices in the group are automatically loaded with a random image fromthat album.

Many-to-one refers to actions, where multiple content/actions areassigned to a single display device. For example, users could drag anddrop an album icon on to a single display device to influence just thatdisplay device, such that all images in the dragged album form a stackand the display device iterates through them with a slideshow.

In one embodiment, there is a clear need for ways in which users mayexpress their intent to select/switch to any of those four groupingmodes before/during their interaction to obtain desired effect/output.In one example, the user may in advance select the grouping mode forconsecutive action to be made in one-to-one mode. Here the user selects,for example, a toggle button to switch to ‘Single’ mode and drags anddrops a filter. As the user drags and hovers a black and white filterover a display in a blue group, only that particular display devicehighlights to show that it will be affected by this action. On drop,only that particular display device gets applied with a black and whitefilter.

In another example, a situation may arise where the user in advanceselects the grouping mode for a consecutive action to be made inone-to-many mode. Here the user selects the toggle button to switch to‘Group’ mode and drags and drops a filter. As the user drags and hoversover a black and white filter over a display in the blue group, alldisplays in the blue group highlight to show who will be affected bythis action. On drop, all displays in blue group get applied with ablack and white filter.

In yet another example, a user selects the grouping mode for consecutiveaction to be made during the drop interaction. As the user drags andhovers a black and white filter over a display device, the overlaysplits into two halves/sub-menus, providing options between ‘single’ and‘group’ mode. If the user drags and hovers over the ‘group’ sub-menu,all other display devices in the same group (e.g., a blue group),highlight to show which would be affected by this action. On the otherhand, if the user drags and hovers to ‘single’ sub-menu, only thatdisplay device becomes highlighted. Depending on which of these twosub-menus the user drops the dragged filter to, the black and whitefilter is applied as one-to-one or one-to-many. Alternatively, the usermay also select the grouping mode for consecutive action as they beginthe drag operation. For example, as the user starts to drag a black andwhite filter, two sub-menus appear near the point of drag. Onerepresents a ‘group’ action and the other represents a ‘single’ action.If the user drags the effect to either sub-menu, that particularsub-menu gets activated as the mode during drop. In FIG. 20A, the userdrags and makes a first stop at single mode sub-menu, activatingone-to-one curation, and then continues on to hover and drop the effecton a display, just affecting that display. If the user drags and makes afirst stop at a group mode sub-menu, activating one-to-many curation,and then continues on to hover and drop the effect on a display devicefrom the blue group, affecting all displays in the blue group.

There are several models of behavior possible within a group:

-   -   Albums may dictate group indications (e.g., color), that is        there is a strict correlation between indication and albums. For        example, all display devices showing pictures from a Christmas        album have to be part of the red group. When a display device/or        the group of display devices is assigned the beach album, they        have to switch to becoming a blue group (e.g., the beach album        is associated with color blue);    -   Display devices may be part of the same group, but do not        necessarily have to show image content from the same album; and    -   Follow everyone else in the group. For example, groups may        retain their color and switch between albums, but all display        devices in the group have to switch to the new album as well        (i.e., all display device show the same album).

FIGS. 17A-B show an example illustration of an example templaterecommendation that measures the display devices and providesrecommendations of various configurations for multiple display devices,according to one embodiment. A ledge, which is a horizontal bar affixedto the wall is used to mount the displays. Electronic device 120 alreadyknows the details of the ledge, such as its dimensions, spacing ofmounting slots etc., and when switched to ledge mode, takes thesedetails into consideration for its calculations. As illustrated, FIG.17A shows display devices 1901, 1902, 1903, 1904, 1905, 1906, 1907 and1908 placed on shelves on a wall. In one example, in a ledge mode, thedisplay devices may be found to not necessarily align well with eachother. In one example, the electronic device 120 captures an image ofthe overall ledge layout. Each display device 1901, 1902, 1903, 1904,1905, 1906, 1907 in the ledge mode is recognized through patternmatching, and is paired with the electronic device 120. In oneembodiment, the electronic device 120 obtains the length and width ofeach display device during the pairing process. In one example, thelength and width of each display may be gathered through a separateprocess. In one example, the pattern matching recognizes three patternsfor the display devices 1901, 1902, 1903, 1904, 1905, 1906, 1907 aspatterns 1920, 1921 and 1922 as shown in FIG. 17A.

As the size of the ledge is fixed and known to the electronic device120, with the knowledge of the total number of display devices, and thesize and shape of each, the electronic device 120 is able to makerecommendations of how to place the display devices in the ledge mode tothe user.

Each assembly manner is referred to as a template. As illustrated inFIG. 17B, every template shows several numbers. Inside the frame of theledge, there is marked measurement 1910 that the user can place (unlesspre-installed) and follow the numbers provided in the template to placethe display devices. As shown, the templates may be selected from atemplate selection portion shown on the electronic device 120. In FIG.17B, the template 1930 is selected and the outline of the displaydevices based on shape are displayed in display 1940 along with thenumbers representing positions or distance/measurement according to themarked measurement 1910. The user simply places the display devicesbased on size/shape at the number on the marked measurement on eachledge according to the template. This assists the user in arranging thedisplay devices (e.g., display devices 1901, 1902, 1903, 1904, 1905,1906, 1907) on the ledge or shelf.

FIGS. 18A-B show an example illustration of an example templaterecommendation for an overall canvas layout 2010 for multiple displaydevices, according to one embodiment. A canvas, which is a flat panelaffixed to the wall, is used to mount the displays. Electronic device120 already knows the details of the canvas, such as its dimensions, thespacing of mounting slots etc., and when switched to canvas mode, takesthese details into consideration for its calculations. In one example, apicture is captured of the overall canvas 2010 layout using electronicdevice 120. The size of the canvas 2010 is fixed and known to theelectronic device 120. The visual app 129 (FIG. 2) may calculate thenumber of possible combinations of placement of the display devicesbased on the size of the canvas 2010, the sizes and shapes of theavailable display devices (e.g., display devices 140, FIG. 2), the powercharging positions of such display devices, the power charging positionsavailable on the canvas 2010, etc. (this information may be retrievedduring the pairing process or through a separate process, for example,discovery of the canvas 2010).

In canvas mode, the template recommendation is not limited to align adisplay device along the horizontal direction as in the ledge mode, butalso along the vertical direction. Each template 2030 may also indicatethe position marks on a display 2040 that match the same marks on thecanvas 2010 to help a user fasten the display devices following thetemplate recommendation. With an easy tool such as this, users caneasily set up a wall of display devices in a few hours with complicatedarrangements such as that in a heart shape, crescent moon shape, etc.,which would have traditionally taken days of planning, measuring,aligning, etc.

In one embodiment, template recommendations may be applied to a tilemode as well. In one embodiment, the electronic device 120 captures animage of the layout of the tiles using her smart phone. The user isallowed to carve out certain areas or tiles from a picture indicatingthat display devices should not be placed at these positions. The visualapp 129 (FIG. 2) then recommends where to place display devices for theremaining available tiles, for the selected content, to achieve the mostaesthetic display effect. In situations where the ledge or canvas usedby user is unknown or when the user mounts on surfaces such as tabletopswhose dimensions are unknown, the system is able to provide options andrecommendations for possible layouts, but would not be able to providespecific measurements or positions to where the display devices 140would need to be placed.

Arrangement recommendations can also be influenced by other factors.Because these display devices are networked, it is possible to thenbring out a social element. In one example, real-time information as towhat are the popular arrangement styles that users across the countryprefer may be collected and recommended to others. On the contrary, ifsomeone prefers a unique arrangement, recommendations for the leastchosen arrangements may be made. Certain arrangements are geometricallysymmetric and are more visually pleasing. These may be made as specialrecommendations for the user. In one embodiment, the arrangements a userhad previously used may be tracked, such that these arrangements may beswitched back to rather easily. This allows for temporary re-arrangementof displays during parties, holiday season, etc., but the arrangementsmay then be placed back exactly how they were before there-arrangements. In one embodiment, the template recommendations aremade based on the content selected to be displayed in the cluster ofdisplays. For example, in special scenarios such as a single image isspread across multiple images randomly, random template arrangements,may affect the aesthetic feel of the image being displayed. The systemat this point can recognize that a single image is being spread acrossmultiple displays and only suggest valid template options orrecommendations for the involved displays.

In one embodiment, in any of the above mentioned arrangement modes, themounting surface (ledge/canvas) may contain LED lights distributedacross the mount. Once calculations are done by the electronic device120 (or separately processed), these LED lights may blink, so the usercan directly place the display device at that location. This removes thenecessity to even look for numbers or a measurement. In anotherembodiment, when these mounting surfaces contain sensors to assessplacement/mounting of display devices, they may also confirm that theuser has performed the right operation. The electronic device 120 maythen notify the user at every stage of accomplishment in performing themounting/arranging operation, guiding them. This significantly reducesthe cognitive load of the user and allows them to enjoy the act ofmounting and arranging display devices.

FIG. 19 shows an example illustration of different viewing perspectivesfor multiple display devices on a wall 2110. As shown, a person 2130standing closer to the wall 2110 has a different viewing perspectivethan a person 2140 standing farther away from the wall 2110. On theright side of FIG. 19, the perspectives of the person 2130 and 2140 arerepresented by point 2131 and 2141, respectively, in a geometricalgraphical representation 2120. The difference in perspectives affectsthe way the wall 2110 of display devices may be perceived.

FIGS. 20A-D show an example illustration of determining the position ofan ideal view point, based on the size, shape and overall layout ofmultiple display devices, according to one embodiment. As shown in FIG.20A, multiple display devices (e.g., display devices 140, FIG. 2) arearranged on a wall 2210. In FIG. 20B, an electronic device 120 capturesa view of the wall 2210. In FIG. 20C the electronic device 120 performsmeasurements to guide users to adjust arrangement of the displaydevices, and the electronic device 120 shows a progress display 2220 asmeasuring. When users are securing the display devices, for example,onto wall 2210, they are close to the display devices such that theycannot have a good view of the overall effect from the usual viewingposition of a distance away from the wall 2210. Additionally, being thatclose provides a very distorted perspective view, which makes it hard toread straight lines and hence estimate how well a display device isaligned with the horizontal, or with respect to another display device.Typically, the ideal spot that provides the best and least distortedviewing experience is some distance away from the wall 2210 and directlyfacing the wall 2210 (e.g., the person 2140, FIG. 19 as compared to theperson 2130). In FIG. 20D, after taking measurements and processing thedisplay devices on the wall 2210, a projected view 2230 is provided,which is a generated image of a view of the wall 2210 from the idealviewing spot or position of the display devices with content displayed.

In one embodiment, the user that is arranging the display devices maytake a picture or a video of all the display devices from anyperspective, using the electronic device 120. A perfectly rectangularobject, for example, may appear deformed when the image is taken from aperspective that is not perpendicular to its plane (e.g., the image onthe electronic device shown in FIG. 20B). Once the display devices arepaired with the electronic device 120 to obtain required informationsuch as the image displayed in each electronic device 120, theirphysical dimensions etc., the sensor(s) of the electronic device 120such as a camera may further determine the distance and angle of theelectronic device 120 to the display devices. Alternatively, as thesizes and shapes of the display devices are known to the electronicdevice 120, the degree of the deformation may be used to calculate therelative position of the user, with respect to the deformed object inthe image. As the user moves closer and further away from a displaydevice 140, the size of its virtual overlay increases and decreasesrespectively. When the user is exactly in front of the display device140, its ideal viewpoint, the virtual overlay associated with displaydevice 140 has edges that are perfectly at 90 degrees to each other. Themore the user moves away from the normal axis sideways or top down, theedges of the virtual overlays stray more from the 90-degree angle tomatch the distorted perspective. This information may be mathematicallygeneralized using principles of projective geometry, homography, etc.,and compared with the known physical dimensions of the display toestimate exactly where, with respect to display device 140, that theuser and the electronic device 120 is currently positioned. Once thecurrent position and viewpoint is calculated, the amount of counterrotation and movement required to obtain the ideal viewpoint is thencalculated.

In one example, the electronic device 120 is able to determine theposition of an ideal view point, based on the size, shape and overalllayout of the display devices. The ideal view point would be a position,and at a reasonable distance from the wall, that the electronic device120 can take an image without causing any deformation. In oneembodiment, the visual app 129 (FIG. 2) may calculate how to counterrotate the perspective and crop the captured image by virtually movingthe electronic device 120 from the real position recognized to the idealview point. The orientation of the image may thus be adjusted as if theimage is captured by the electronic device 120 at the ideal view point.This way, the user can follow the cropped image to adjust thearrangement of the display devices without requiring additional helpfrom another user, or constantly moving to the ideal viewing spot tocheck on the display effect. In one embodiment, the electronic device120 allows the viewer to view not just from this ideal viewpoint, butfrom any angle and distance with respect to the display devices, aspermissible by the resolution of the image. For example, the user canidentify a preferred viewpoint, which is not the ideal view point. Thecropped image will allow the user to see the effect of the arrangementof the display devices from the preferred viewpoint. In one embodiment,the preferred viewpoint may be determined by visual app 129, based onuser preference learned from previous arrangements, and/or environmentaldata surrounding the display devices (e.g., size of the display space,furniture and/or plants around the display devices that may block theview of the display devices, areas that have the line sight to thedisplay devices that the user frequently presides, etc.).

FIGS. 21A-D show an example illustration of capturing a frame, analyzingthe prominent colors in a room 2310 and displaying these colors as ascrollable list 2320 and presenting images 2330/2331 based on the list2320 on an electronic device 120, according to one embodiment. In oneexample, the detected ambient colors of a room 2310 may be used torecommend images/art work from either the user's collection (e.g.,gallery, albums, etc.) or from curated services, that best match thecolor palette of the ambience in the room 2310. In one example, FIG. 21Ashows a user pointing the electronic device 120 camera 128 (FIG. 2) attheir room 2310. The electronic device 120 captures a frame, analyzesthe prominent colors in the room 2310 and displays them as a scrollablelist 2320 at the bottom of the screen. When the user presses on any ofthe colors (e.g., color 2321) in the list 2320, the visual app 129 thensifts through its database, looking for images which have this selectedcolor as its dominant color and presents them to the user as ascrollable collection of images 2330 as shown in FIG. 21C. Similarly, aselection of color 2321 results in a collection of images 2331 in FIG.21C. The user can select an image of interest and use this for curationof images for the display devices on the wall 2310.

FIGS. 22A-C show an example illustration of assessing the color of awall 2410 surrounding a display device frame and using that informationto color match the mat 2420 of the display device, according to oneembodiment. Most picture frames available on the market come with a mat.Mats usually serve the purpose of bringing attention to the picture byproviding color relief and separation from the plethora of informationsurrounding the frame. These mats are mostly made of paper, but do alsocome in a variety of textures and colors. Some picture frames havetransparent mats, which are see through and pick up the color of thewall behind it. With digital display devices, one embodiment simulatesthe experience of a mat using digital mats, which serve the same purposeas their analog picture frame counter parts.

In one embodiment, a computer vision technique is used to assess thecolor of the wall 2410 surrounding a display device and using thatinformation to color the mat 2420. In one example, the electronic device120 captures the image of the wall 2410, analyzes the color componentsof the wall 2410, and provides a selection of mats 2430 matching thewall 2410 color(s) in FIG. 22B. In FIG. 22C, a selection of a mat fromthe selection of mats 2430 is made and the mat 2420 of the centerdisplay device is changed to mat 2421.

FIGS. 23A-C show an example illustration of identifying repeatingpatterns on a wall 2510, and digitally synthesizing the patterns toprovide a similar transparent mat effect for a display device, accordingto one embodiment. In one example, an effect achieved is similar to thatof having a transparent mat (e.g., clear or see through glass) aroundphoto/video content on the display device of the wall 2510. Walls withpatterned designs such as tiles, bricks and wallpaper are popular inmany parts of the world. One embodiment may be extended to thesepatterned walls. If the repeating patterns are small enough, a singleimage is captured that contains at least a few examples of the pattern,fully visible and unoccluded by other objects. The system thenidentifies this repeating pattern and stores it for future processing.If the repeating patterns are too large, or the room is setup in amanner that the fully visible examples are far from each other, the usercan achieve a similar effect by stitching together multiple overlappingimages to form a single large image. A step is performed, where multipleimages of the wall 2510 on which the display device is mounted. Theseimages are then processed using computer vision and stitched together toform one large image. In one embodiment, if the electronic devicesupports it, a single large image covering the display device (and anyother display devices) and a substantial portion of the wall 2510 willalso do. A process analyzes the stitched image to identify repeatingpatterns, their arrangement and spatial relationship to the location ofthe display device (e.g., pattern matching using instance-level objectdetection, category-level object recognition, block-based visual-patternmatching, etc.). In one embodiment, the system runs a process thatperforms the following: a) the image is analyzed to identify a patternthat is clearly repeating across the image; b) the identified pattern isthen separated via cropping and stored in a database for reference alongwith other meta data (referred to herein as the reference pattern); c)the immediate borders surrounding each display device 140 is thenanalyzed for presence of patterns; d) these border patterns are thenchecked to see if they are a subset of the identified reference pattern;e) once there is a match, then it is determined that the detected borderpattern is actually larger, consisting of all of the information as thatof the reference pattern, and portions of it being occluded by thedisplay device 140; f) subsequently, the regions of the pattern thatwould have been visible but are currently occluded by display device140's mat 2520 area are digitally synthesized via region filling andrendered in the accurate location. FIG. 23C shows the result mat 2521matching the pattern on the wall 2510 to provide a transparent mateffect.

In another example, a wall may have a brick texture pattern. In thisexample, a display device on the wall originally has a white mat. Inthis example, the user desires to have a clear mat effect on the displaydevice. An electronic device 120 captures an image of the wall andanalyzes the wall pattern to simulate a clear mat such that the patternof the wall continues in the mat region of the display device. The matis changed to a mat that appears to be clear as the pattern of the wallcontinues into the mat region.

When the user has the ability to capture an image of the wall withoutthe display devices 140, either prior to mounting or is able totemporarily remove and mount the display devices 140, another example isdescribed below. The underlying technology may be thought of as asubtraction mechanism. In one embodiment, an image A of the wall iscaptured without the display devices 140. Then the display devices 140and the respective images are placed at their correct locations andanother image B is captured. Computer vision technology is used to alignthe two images A and B, and a one-on-one comparison is processed tofigure out regions C that are visible in image A, but are being occludedin image B. The respective pixels from C that lie on the mat area ofeach of these display devices 140 are then determined. Texturecoordinates for these regions are calculated and then sent to therespective display devices, which then display content from originalimage A in the mat area to provide an illusion of a clear mat. Thisembodiment is especially useful in scenarios when every instance of therepeating pattern is occluded in one way or the other, making it hardfor other embodiment processing to execute. This technique is also veryvaluable in scenarios where instead of a repeating pattern, the wall isdecorated with one large design such as a photo, painting or mural wherethe above embodiment is not usable due to absence of a repeatingpattern.

FIGS. 24A-C show an example of customizing matting for imaging on adisplay device, according to an embodiment. In one embodiment, controlis provided to the user as to how much of a mat they would like to see.In the example shown in FIG. 24A, the electronic device 120 camera 128(FIG. 2) is pointed at a display device 2610 having a mat 2615. An outermat control interface 2620 and an inner mat control interface 2630 aredisplayed on the electronic device 120. In the example shown in FIG.24B, the inner mat control interface 2630 is used to adjust thethickness of the mat. In this example, the mat 2615 thickness betweenthe four images is increased, resulting in new mat 2616. In the exampleshown in FIG. 24C, the outer mat control interface 2620 is used toincrease the thickness of the mat 2616, resulting in the new mat 2617.It should be noted that the mat thickness may be decreased as well ifdesired by adjusting the scroll bars for the outer mat control interface2620 and the inner mat control interface 2630.

FIGS. 25A-B shows another example of customizing matting for imaging ona display device, according to an embodiment. FIG. 25A shows an examplewhere the electronic device 120 camera 128 (FIG. 2) is pointed at adisplay device 2710 having a mat 2715. As shown in FIG. 25B, the mat maybe touched and dragged from the corner 2720 either inwards or outwardsto increase, decrease or remove altogether the mat 2715. In thisexample, the mat 2715 has its thickness increased resulting in mat 2716shown on the display 2710.

FIG. 26 shows a block diagram of a process 2800 for overlaying visualeffects on a captured image, according to one embodiment. In oneembodiment, in block 2810 process 2800 provides, when the electronicdevice is pointed at a display device (e.g., a display device 140),capturing of an image displayed in the display device on an electronicdevice (e.g., with a camera 128 of electronic device 120, FIG. 2). Inblock 2820, a pattern of the image is matched in a pattern matchingdatabase (e.g., using a matching processor (e.g., processor 3111, FIG.29) or process (e.g., visual app 129, FIG. 2 or content curation anddisplay management processing 3130), which is part of the AR library(e.g., part of visual app 129 and/or applications 1-N 127), resides onelectronic device 120 and is trained to detect a finite set of images).In block 2830, if the image is present in the pattern matching database,the details are provided to an AR library, such as presence of an image,the name of the image identified, etc. In block 2840 process 2800detects where in a scene the pattern is present and the location of thefour corners of the pattern. In one example, the AR library uses itscomputer vision process or processor to determine where in the scenethis pattern is present and the details. In block 2850, visual effectsare overlayed within the detected four corners for providing a virtualoverlay that remains locked to a display device of the electronicdevice. In one example, the AR library uses its computer graphicsprocess or processor to overlay desired visual effects within thedetected four corners, that provide the illusion of a virtual overlaythat remains locked to the display device.

FIG. 27 shows a block diagram of a process 2900 for spreading an imageover multiple display devices, according to one embodiment. In oneembodiment, block 2910 provides determining relative positions of eachof multiple display devices (e.g., display devices 901, 902 and 903,FIGS. 7A-7D) with respect to each other and with respect to anelectronic device (e.g., electronic device 120). In block 2920 process2900 provides determining aspect ratios and physical dimensions for eachof the multiple display devices to subtend a particular sub-region of animage to display. In block 2930 process 2900 provides for associatingcorresponding texture coordinates for each of the multiple displaydevices. The texture coordinates indicating a portion of the image todisplay on each of the multiple display devices. In block 2940 process2900 provides for sending each of the multiple display devices a messageincluding the image and associated texture coordinates. In block 2950process 2900 provides causing loading of the image across the multipledisplay devices based on the associated texture coordinates.

FIG. 28 shows a block diagram of a process 3000 for managing displaydevices, according to one embodiment. In one embodiment, in block 3010process 3000 receives an image including one or more display devices(e.g., display devices 140, FIG. 2). The image includes contextual dataassociated with environmental information surrounding the one or moredisplay devices with respective first content displayed. In block 3020process 3000 displays an updated image of the image of the one or moredisplay devices. The updated image projects a view of the one or moredisplay devices from a desired perspective.

In one embodiment, process 3000 may further include recognizing a devicetype for each of the one or more display devices based on a respectivecontent displayed. Measurement data is retrieved for each of the one ormore display devices according to corresponding device type. One or moredevice arrangement templates is provided. Each template providesindicators for mapping to marked measurement of the one or more displaydevices.

In one embodiment, process 3000 may further include recommending afilter based on the contextual data and the environmental information.The filter is configured to neutralize a color palette. A mat of atleast one display device is caused to be updated based on the contextualdata and the environmental information.

In one embodiment, process 3000 may further include sensing, by sensorsof an electronic device (e.g., electronic device 120, FIG. 2), datacomprising distance and angle of the electronic device to the one ormore display devices. The deformation level is determined based on thesensed data and image of the display devices. Position of a viewpoint iscalculated relative to the one or more display devices based on thedeformation level. An ideal viewing position is determined based onsize, shape and layout of the one or more display devices.

In one embodiment, process 3000 may include determining combinations ofplacement locations for the one or more display devices. Display devicerepresentations are marked with measurement information for a displaydevice placement structure. Template recommendations are displayed withthe measurement information based on the determined combinations ofplacement locations.

In one embodiment, process 3000 may additionally include detectingambient colors and lighting of an environment surrounding the one ormore display devices. Filters for neutralizing a color palette ofcontent displayed on the one or more display devices is determined basedon the detected ambient colors and lighting. Recommendations areprovided for application of the determined filters to the contentdisplayed on the one or more display devices.

In one embodiment, process 3000 may also include detecting ambientcolors and lighting of an environment surrounding the one or moredisplay devices. A group of content for display on the one or moredisplay devices may be determined based on the detected ambient colorsand lighting.

In one embodiment, process 3000 may further include detecting a patterndesign on an environment surrounding the one or more display devices.The pattern design is analyzed to for identifying a repeating pattern,arrangement of the repeating pattern and spatial relationship of therepeating pattern to a location of at least one of the one or moredisplay devices. Filling patterns are determined for regions of therepeating pattern that would display under a mat area of the at leastone display device. The mat area content is replaced with the fillingpatterns.

In one embodiment, process 3000 may further include determiningcombinations of placement locations for the one or more display devices.Template recommendations are displayed based on the determinedcombinations of placement locations. Indications are provided on acanvas structure for display device placement based on a receivedselection of one of the template recommendations.

FIG. 29 is a high-level block diagram showing an information processingsystem comprising a computing system 3100 implementing one or moreembodiments. The system 3100 includes one or more processors 3111 (e.g.,ASIC, CPU, etc.), and may further include an electronic display device3112 (for displaying graphics, text, and other data), a main memory 3113(e.g., random access memory (RAM), cache devices, etc.), storage device3114 (e.g., hard disk drive), removable storage device 3115 (e.g.,removable storage drive, removable memory, a magnetic tape drive,optical disk drive, computer-readable medium having stored thereincomputer software and/or data), user interface device 3116 (e.g.,keyboard, touch screen, keypad, pointing device), and a communicationinterface 3117 (e.g., modem, wireless transceiver (such as Wi-Fi,Cellular), a network interface (such as an Ethernet card), acommunications port, or a PCMCIA slot and card).

The communication interface 3117 allows software and data to betransferred between the computer system and external devices through theInternet 3150, mobile electronic device 3151, a server 3152, a network3153, etc. The system 3100 further includes a communicationsinfrastructure 3118 (e.g., a communications bus, cross bar, or network)to which the aforementioned devices 3111 through 3117 are connected.

The information transferred via communications interface 3117 may be inthe form of signals such as electronic, electromagnetic, optical, orother signals capable of being received by communications interface3117, via a communication link that carries signals and may beimplemented using wire or cable, fiber optics, a phone line, a cellularphone link, a radio frequency (RF) link, and/or other communicationchannels.

In one implementation of one or more embodiments in a mobile wirelessdevice (e.g., a mobile phone, tablet, wearable device, etc.), the system3100 further includes an image capture device 3120, such as a camera 128(FIG. 2), and an audio capture device 3119, such as a microphone 122(FIG. 2). The system 3100 may further include application processing orprocessors as MMS 3121, SMS 3122, email 3123, social network interface(SNI) 3124, audio/video (AV) player 3125, web browser 3126, imagecapture 3127, etc.

In one embodiment, the system 3100 includes content curation and displaymanagement processing 3130 that may implement processing similar asdescribed regarding visual app 129 (FIG. 2) and processing contentcuration, display device management as described above. In oneembodiment, the content curation and display management processing 3130along with an operating system 3129 may be implemented as executablecode residing in a memory of the system 3100. In another embodiment, thecontent curation and display management processing 3130 may be providedin hardware, firmware, etc.

One or more embodiments provide intuitive and easy curation of displaydevices (e.g., display devices 140, FIG. 2) in a visual manner, and mayfunction by tracking the images in the display device itself. The one ormore embodiments do not require additional visual markers such as ARpatterns, bar codes and QR codes. Advanced interactions, such as dragand drop are employed to assign content to display devices. Applyingspecial effects to specific display devices is provided via advancedspatial interactions. Spreading an image across multiple display devicesare provided in a spatially coherent manner. Selecting importantregions/features of interest in an image to be displayed to specificdisplay devices is provided. The one or more embodiments enablespatially intelligent editing of images based on the size and aspectratio of the display device that the image is going to be assigned to.

The one or more embodiments provide virtual representations of a clusterof display devices as virtual walls, which may be saved, retrieved andcurated even when geographically separated from the physical displayswith the same user experience. Dynamic creation of groups of displaysand their characteristics/behavior is provided. Several types of groupinteractions via unique UI mechanisms are provided. Suggestions forselection of filters based on neighboring image content and ambience ofthe display devices such as physical wall color and lighting isprovided. Suggestions for the most suitable photos/content either from alocal collection or curated services based on ambient information isprovided.

The one or more embodiments provide suggestions of various templates andpatterns of arrangement by detecting the number of display devices to bemounted and their various shapes and sizes, and perform calculations andprovide precise measurements to achieve a selected pattern. Real-timeposition and alignment feedback is provided and corrections andguidelines are also provided in order to achieve a selected pattern. Afar-away perspective of a cluster of displays is provided, even whenstanding close by to them, which assists with decisions aboutpositioning and alignment. The environment is analyzed to mimic itsfeatures such as colors, texture and patterns within digital matregions, resulting in blending of the digital frames with theenvironment and pleasing visual effects.

As is known to those skilled in the art, the aforementioned examplearchitectures described above, according to said architectures, can beimplemented in many ways, such as program instructions for execution bya processor, as software modules, microcode, as computer program producton computer readable media, as analog/logic circuits, as applicationspecific integrated circuits, as firmware, as consumer electronicdevices, AV devices, wireless/wired transmitters, wireless/wiredreceivers, networks, multi-media devices, etc. Further, embodiments ofsaid Architecture can take the form of an entirely hardware embodiment,an entirely software embodiment or an embodiment containing bothhardware and software elements.

One or more embodiments have been described with reference to flowchartillustrations and/or block diagrams of methods, apparatus (systems) andcomputer program products according to one or more embodiments. Eachblock of such illustrations/diagrams, or combinations thereof, can beimplemented by computer program instructions. The computer programinstructions when provided to a processor produce a machine, such thatthe instructions, which execute via the processor create means forimplementing the functions/operations specified in the flowchart and/orblock diagram. Each block in the flowchart/block diagrams may representa hardware and/or software module or logic, implementing one or moreembodiments. In alternative implementations, the functions noted in theblocks may occur out of the order noted in the figures, concurrently,etc.

The terms “computer program medium,” “computer usable medium,” “computerreadable medium”, and “computer program product,” are used to generallyrefer to media such as main memory, secondary memory, removable storagedrive, a hard disk installed in hard disk drive. These computer programproducts are means for providing software to the computer system. Thecomputer readable medium allows the computer system to read data,instructions, messages or message packets, and other computer readableinformation from the computer readable medium. The computer readablemedium, for example, may include non-volatile memory, such as a floppydisk, ROM, flash memory, disk drive memory, a CD-ROM, and otherpermanent storage. It is useful, for example, for transportinginformation, such as data and computer instructions, between computersystems. Computer program instructions may be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

Computer program instructions representing the block diagram and/orflowcharts herein may be loaded onto a computer, programmable dataprocessing apparatus, or processing devices to cause a series ofoperations performed thereon to produce a computer implemented process.Computer programs (i.e., computer control logic) are stored in mainmemory and/or secondary memory. Computer programs may also be receivedvia a communications interface. Such computer programs, when executed,enable the computer system to perform the features of the embodiments asdiscussed herein. In particular, the computer programs, when executed,enable the processor and/or multi-core processor to perform the featuresof the computer system. Such computer programs represent controllers ofthe computer system. A computer program product comprises a tangiblestorage medium readable by a computer system and storing instructionsfor execution by the computer system for performing a method of one ormore embodiments.

Though the embodiments have been described with reference to certainversions thereof; however, other versions are possible. Therefore, thespirit and scope of the appended claims should not be limited to thedescription of the preferred versions contained herein.

What is claimed is:
 1. A method for device management, comprising:receiving an image including one or more display devices, the imageincluding contextual data associated with environmental informationsurrounding the one or more display devices; and controlling, by aprocessor device, a display showing an updated image of the image of theone or more display devices, the updated image projecting a view of theone or more display devices from a desired viewing perspective usingperspective correction.
 2. The method of claim 1, further comprising:recognizing a device type for each of the one or more display devicesbased on a respective content displayed; retrieving measurement data foreach of the one or more display devices according to correspondingdevice type; and providing one or more device arrangement templates,each template providing indicators for mapping to marked measurement ofthe one or more display devices.
 3. The method of claim 1, furthercomprising: recommending a filter based on the contextual data and theenvironmental information, wherein the filter is configured toneutralize a color palette; and causing update of a mat of at least onedisplay device of the one or more display devices based on thecontextual data and the environmental information.
 4. The method ofclaim 1, further comprising: sensing, by sensors of an electronicdevice, data comprising distance and angle of the electronic device tothe one or more display devices; determining a deformation level basedon the sensed data and image of the one or more display devices;calculating position of a viewpoint relative to the one or more displaydevices based on the deformation level; and determining an ideal viewingposition based on size, shape and layout of the one or more displaydevices.
 5. The method of claim 1, further comprising: determiningcombinations of placement locations for the one or more display devices;marking display device representations with measurement information fora display device placement structure; and displaying templaterecommendations with the measurement information based on the determinedcombinations of placement locations.
 6. The method of claim 1, furthercomprising: detecting ambient colors and lighting of an environmentsurrounding the one or more display devices; determining filters forneutralizing a color palette of content displayed on the one or moredisplay devices based on the detected ambient colors; and providingrecommendations for application of the determined filters to the contentdisplayed on the one or more display devices.
 7. The method of claim 1,further comprising: detecting ambient colors and lighting of anenvironment surrounding the one or more display devices; determining agroup of content for display on the one or more display devices based onthe detected ambient colors and lighting.
 8. The method of claim 1,further comprising: detecting a pattern design on an environmentsurrounding the one or more display devices; analyzing the patterndesign for identifying a repeating pattern, arrangement of the repeatingpattern and spatial relationship of the repeating pattern to a locationof at least one of the one or more display devices; determining fillingpatterns for regions of the repeating pattern that would display under amat area of the at least one display device; and replacing the mat areacontent with the filling patterns.
 9. The method of claim 1, furthercomprising: determining combinations of placement locations for the oneor more display devices; displaying template recommendations based onthe determined combinations of placement locations; and providingindications on a canvas structure for display device placement based ona received selection of one of the template recommendations.
 10. Asystem for device management comprising: a memory storing instructions;and at least one processor executing the instructions including aprocess configured to: receive an image including one or more displaydevices, the image including contextual data associated withenvironmental information surrounding the one or more display devices;and control a display showing an updated image of the image of the oneor more display devices, the updated image projecting a view of the oneor more display devices from a desired viewing perspective usingperspective correction.
 11. The system of claim 10, wherein the processis further configured to: recognize a device type for each of the one ormore display devices based on a respective content displayed; retrievemeasurement data for each of the one or more display devices accordingto corresponding device type; and provide one or more device arrangementtemplates, each template providing indicators for mapping to markedmeasurement of the one or more display devices.
 12. The system of claim10, wherein the process is further configured to: recommend a filterbased on the contextual data and the environmental information, whereinthe filter is configured to neutralize a color palette; and cause anupdate of a mat of at least one display device of the one or moredisplay devices based on the contextual data and the environmentalinformation.
 13. The system of claim 10, further comprising: one or moresensors configured to sense data comprising distance and angle of to theone or more display devices; wherein the process is further configuredto: determine a deformation level based on the sensed data and image ofthe one or more display devices; calculate position of a viewpointrelative to the one or more display devices based on the deformationlevel; and determine an ideal viewing position based on size, shape andlayout of the one or more display devices.
 14. The system of claim 10,wherein the process is further configured to: determine combinations ofplacement locations for the one or more display devices; illuminatingdisplay device representations based on measurement information for adisplay device placement structure; and display template recommendationswith the measurement information based on the determined combinations ofplacement locations.
 15. The system of claim 10, wherein the process isfurther configured to: detect ambient colors and lighting of anenvironment surrounding the one or more display devices; determinefilters for neutralizing a color palette of content displayed on the oneor more display devices based on the detected ambient colors; andprovide recommendations for application of the determined filters to thecontent displayed on the one or more display devices.
 16. The system ofclaim 10, wherein the process is further configured to: detect a patterndesign on an environment surrounding the one or more display devices;analyze the pattern design to identify a repeating pattern, arrangementof the repeating pattern and spatial relationship of the repeatingpattern to a location of at least one of the one or more displaydevices; determine filling patterns for regions of the repeating patternthat would display under a mat area of the at least one display device;and replace the mat area content with the filling patterns.
 17. Anon-transitory processor-readable medium that includes a program thatwhen executed by a processor device performs a method comprising:receiving an image including one or more display devices, the imageincluding contextual data associated with environmental informationsurrounding the one or more display devices; and controlling, by theprocessor device, a display showing an updated image of the image of theone or more display devices, the updated image projecting a view of theone or more display devices from a desired viewing perspective usingperspective correction.
 18. The non-transitory processor-readable mediumof claim 17, further comprising: recognizing a device type for each ofthe one or more display devices based on a respective content displayed;retrieving measurement data for each of the one or more display devicesaccording to corresponding device type; and providing one or more devicearrangement templates, each template providing indicators for mapping tomarked measurement of the one or more display devices.
 19. Thenon-transitory processor-readable medium of claim 17, furthercomprising: recommending a filter based on the contextual data and theenvironmental information, wherein the filter is configured toneutralize a color palette; and causing update of a mat of at least onedisplay device of the one or more display devices based on thecontextual data and the environmental information.
 20. Thenon-transitory processor-readable medium of claim 17, furthercomprising: sensing, by sensors of an electronic device, data comprisingdistance and angle of the electronic device to the one or more displaydevices; determining a deformation level based on the sensed data andimage of the one or more display devices; calculating position of aviewpoint relative to the one or more display devices based on thedeformation level; and determining an ideal viewing position based onsize, shape and layout of the one or more display devices.